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Abstract 

The first part of Hilbert's sixteenth problem deals with the classi- 
fication of the isotopy types realizable by real plane algebraic curves 
of given degree m. For m > 8, one restricts the study to the case of 
the M-curves. For m = 9, the classification is still wide open. We say 
that an M-curve of degree 9 has a deep nest if it has a nest of depth 
3. In the present paper, we prohibit 10 isotopy types with deep nest 
and no outer ovals. 

1 Introduction 

Let A be a real algebraic non-singular plane curve of degree m. Its complex 
part CA C CP 2 is a Riemannian surface of genus g = (m — l)(m — 2)/2; 
its real part MA C MP 2 is a collection of L < g + 1 circles embedded in 
MP 2 . If L = g + 1, we say that A is an M-curve. A circle embedded in MP 2 
is called oval or pseudo-line depending on whether it realizes the class 
or 1 of .Hi (MP 2 ). If m is even, the L components of MA are ovals; if m is 
odd, MA contains exactly one pseudo-line, which will be denoted by J . An 
oval separates MP 2 into a Mobius band and a disc. The latter is called the 
interior of the oval. An oval of MA is empty if its interior contains no other 
oval. One calls outer oval an oval that is surrounded by no other oval. Two 
ovals form an injective pair if one of them lies in the interior of the other one. 
We call nest of depth d + 1 a configuration of ovals (Oq, 0±, . . . , Od) such 
that Oi lies in the interior of Oj for all pairs i,j, with j > i. An M-curve 
of degree m = 2k or m = 2k + 1 is said to have a deep nest if it has a nest 
of depth k — 1. Notice that for a curve with deep nest, all of the other ovals 
are empty by Bezout's theorem with an auxiliary line. 

Let us call the isotopy type of MA C MP 2 the real scheme of A; it 
will be described with the following notation due to Viro. The symbol {J) 
stands for a curve consisting in one single pseudo-line; (n) stands for a 
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curve consisting in n empty ovals. If X is the symbol for a curve without 
pseudo-line, 1{X) is the curve obtained by adding a new oval, containing 
all of the others in its interior. Finally, a curve which is the union of 2 
disjoint curves (A) and (B), having the property that none of the ovals of 
one curve is contained in an oval of the other curve, is denoted by (AJIB) .The 
classification of the real schemes which are realizable by M-curves of a given 
degree in MP 2 is part of Hilbert's sixteenth problem. This classification is 
complete up to degree 7 and almost complete in degree 8. A systematic 
study of the case m = 9 has been done, the main contribution being due to 
A. Korchagin. See e.g. [5], [7], [8], [9], [11] for the constructions, and [5], [6], 
PQ) [3], [32], [13] for the restrictions. The main result of the present paper 
is the prohibition of 10 new schemes. The proof is an improvement of the 
classical restriction method: we use Bezout's theorem with auxiliary pencils 
of rational cubics, and add to the classical theorems on complex orientations 
the newer formulas of Orevkov [10] for M-curves with deep nests. 

Let us briefly recall some facts about complex orientations. The complex 
conjugation conj of CP 2 acts on CA with MA as fixed points sets. Thus, CA\ 
MA is connected, or splits in 2 homeomorphic halves which are exchanged 
by conj. In the latter case, we say that A is dividing. Let us now consider a 
dividing curve A of degree m, and assume that CA is oriented canonically. 
We choose a half CA + of CA \ MA. The orientation of CA + induces an 
orientation on its boundary MA. This orientation, which is defined up to 
complete reversion, is called complex orientation of A. One can provide all 
the injective pairs of MA with a sign as follows: such a pair is positive if and 
only if the orientations of its 2 ovals induce an orientation of the annulus 
that they bound in MP 2 . Let II + and II_ be the numbers of positive and 
negative injective pairs of A. If A has odd degree, each oval of MA can be 
provided with a sign: given an oval O of MA, consider the Mobius band 
Ai obtained by cutting away the interior of O from MP 2 . The classes [O] 
and [2J7] of H\{Ai) either coincide or are opposite. In the first case, we say 
that O is negative; otherwise O is positive. Let A + and A_ be respectively 
the numbers of positive and negative ovals of MA. The complex scheme of 
A is obtained by enriching the real scheme with the complex orientation: 
let e.g. A have real scheme {J II 1(a) II (3). The complex scheme of A is 
encoded by (J II l e (a+ II aJ) II /3 + II 0J) where e € {+, — } is the sign of 
the non-empty oval; a+,a_ are the numbers of positive and negative ovals 
among the a; /3_ are the numbers of positive and negative ovals among 
the f3 (remember that all signs are defined with respect to the orientation 
of the pseudo-line J'). 
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Rokhlin-Mishachev formula: If m = 2k + 1, then 

2(n+ - n_) + (A+ - a_) = l - 1 - fc(fc + 1) 

Fiedler theorem: Let C t = {L t ,t G [0, 1]} 6e a pencil of real lines based 
in a point P o/ MP 2 . Consider two lines and Lt 2 of Ct, which 
are tangent to WA in two points P\ and P2, such that P\ and P2 are 
related by a pair of conjugated imaginary arcs in CA n (\jL t ). 

Orient L tl coherently to MA in Pi, and transport this orientation 
through Ct to L t2 . Then this orientation of L t , 2 is compatible to that 
ofRA in P 2 . 

Orevkov formulas for odd degree Let A be an M-curve of degree m = 
2k + 1, with a deep nest. Let 1+ and I- be respectively the numbers of 
positive and negative non-empty ovals; X + and A_ be respectively the 
numbers of positive and negative empty ovals. Let irf,S,s G {+, — } 
be the number of pairs (0,o) where o is an empty oval surrounded by 
O and (S,s) are the signs of(0,o). Then: 

vrl - vr+ = (Z+) 2 , and vr; - vrl + (A+ - A_)/2 = (Z_) 2 + Z_ 

2 Restrictions 

2.1 Definitions and results 

Let Cg be an M-curve of degree 9. Given an empty oval X of Cg, we often 
will have to consider one point chosen in the interior of X. For simplicity, 
we shall call this point also X. We denote the pencil of lines based in X 
by Tx- Let X, Y be 2 empty ovals of Cg. We shall denote by [XY] (resp. 

the segment of line XY that cuts J an even (resp. an odd) number of 
times. We say that [XY] is the principal segment determined by X, Y. Let 
X,Y,Z be three empty ovals of Cg. Corresponding three points X, Y and 
Z determine 4 triangles of MP 2 . We will call principal triangle and denote 
by XYZ the triangle bounded by the segments [XY], [YZ] and [XZ]. We 
denote respectively by Tx '■ Y — > Z and T' x :Y^Z the pencils of lines 
based in X that sweep out the segments [YZ] and [YZ]'. We denote by 
T e x : Y — > Y e G {±} the complete pencils rotating in either direction, 
e = + being chosen for the counter-clockwise direction. If Y\,...,Y n are 
empty ovals met successively by T x : Y\ — > Y\, we say that this pencil 
has one J -jump between 2 consecutive ovals, say Y$, Y^+i if T x : Y^ — > Yi + 
sweeps out the segment [liYi + i]'. Notice that the total number of J'-jumps 
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Figure 1: Oi-jump 



of a complete pencil over a sequence of ovals is always odd and does not 
depend on the choices of e and Y±. Denote by (L t ),t € [0,1] the lines of 
the pencil T e x : Y\ — > Y n . We say that Yi,...,Y n form a Fiedler chain with 
respect to this pencil if each Yi,i = 1, . . . , n — 1 is connected to Yi + \ by a 
pair of imaginary arcs of CAD ({J Lt). Notice that by Fiedler's theorem, the 
signs of Y,i and Y,i + \ coincide if and only if the pencil : Y; L — > Yj + i has a 
,7-jump. 

An ordered group of empty ovals Ai,..., A n of Cg lies in a convex po- 
sition if for each triple A^, Aj, A^, the principal triangle A^AjA^ does not 
contain any other oval of the group and Ai, . . . , A n are the successive vertices 
of (J AiAjAk (the convex hull of the group). 

Let Cg be an M-curve with real scheme (J II a II l(/3 II 1(7)}). We shall 
call the ovals of the groups a, (3 and 7 respectively outer, median and inner 
ovals. Notice that the inner ovals must lie in convex position. Otherwise, 
let A, B, C, D be 4 inner ovals such that D lies inside of the principal tri- 
angle ABC .Then, for any choice of a fifth empty oval E, the conic through 
A, B, C, D, E cuts Cg in more than 18 points. This is a contradiction. 

Definition 1 We say that Cg has a Oi-jump if there exist 2 median ovals 
A, C and 2 inner ovals B,D, such that the line AC separates B from D in 
lnt{0{). 

Notice, again with auxiliary conies, that if Cg has a Oi-jump determined 
by A, B, C, D, then the principal segment [AC] cuts 0±, as shown in Figure 1. 
The lines (AB), (BC), (CD), (DA) give rise to 3 quadrangles and 4 triangles. 
All of the remaining empty ovals lie in the union of the 4 triangles. 
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Figure 2: Cg with J = 1 



Lemma 1 If Cg has a median oval A\ such that the pencil Ta x sweeping 
out 0\ meets successively empty ovals A2, A3, . . . , A2J+2, where the Ai, i 
even (resp. odd) are inner (resp. median) ovals. Then: 

(1) The Ai i = 1, . . . , 2 J + 2 lie in convex position. 

(2) Let Ai, Aj, Ak, Ai be 4 consecutive ovals in the cyclic ordering, we 
denote by Tj the triangle determined by the lines (AiAj)(Aj Ak)(AkAi) with 
edge [AjA^], that does not intersect the convex hull of A±, A2, . . . , Au+i- 
All of the remaining empty ovals lie in {jTj, j = 1, . . . , 2 J + 2; if an oval A 
lies in Tj n Int{02), A is not separated in Tj from the edge [AjA^] by J . 

(3) There is a natural cyclic ordering of the empty ovals. This ordering 
is given by: the complete pencils of lines J-b, for any inner oval B, and the 
pencils J- a, where A is a median oval in some triangle T, sweeping out the 
2 J triangles Tj having no common median vertex with T. The ovals in the 
triangles T\,T2,... ,T2,j+2 appear successively in the cyclic ordering. 

(4) Ifa = 0, then A+ - A_ = 

Proof The points (1), (2), (3) are easily proven using Bezout's theorem 
with conies. Notice that if a = 0, the pencils of lines J : a 1 '■ Aj — > Tj — > A^ 
have no ^7-jumps, and give rise to a closed Fiedler chain involving all of the 
empty ovals, hence (4) follows. □ 

Definition 2 Let Cg be an M-curve with deep nest, and no outer ovals. 
The curve Cg has J Oi-jumps with distribution (l\, . . . , I2J+2) if Cg verifies 
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the condition of Lemma 1 and = 1, . . . , 2 J + 2 are the cardinals of the 
successive groups of inner and median ovals in the cyclic ordering. 

The case J = 1 is illustrated in Figure 2. Notice that the number of 
Oi-jumps and their distribution is a rigid isotopy invariant of Cg. 

Theorem 1 The real schemes {J II l{/3 II 1(7))), with (3 + 7 = 26 and /?, 7 
odd are not realizable by M-curves of degree 9. 

Among the 13 admissible schemes of that form, the 3 with = 1, 3 and 
25 had already been forbidden in [BJ. 

Theorem 2 Let Cg be an M-curve with real scheme ( II 1 (y3 II 1 (7) ) ) , with 
even (3, 7. Then: 

1. if Cg has Oi- jumps, then the complex scheme of Cg is 
(J II 1_<(V)+ n 11 H ( V)-»> 

(j 11 1+((^)+ h (^)_ n i+(( V)+ u (^)->»- 

otherwise, the complex scheme of Cg is 

(j n i-((^)+ n (^*)_ n i_((|) + n ©_))). 

Notice that by Theorem 2, the real scheme (J' II 1(2 II 1(24))) is not 
realizable. This real scheme was already forbidden in [6j. S. Orevkov con- 
structed M-curves realizing the 10 real schemes (^711 1 (/3 II 1(7))), with /3, 7 
even, with 4 < [3 < 22. Each of these curves has 3 Oi-jumps, they realize 
the complex schemes {J II l+((^)+ II (^)_ II l+((^)+ U (^)-))) 
(private communication) . 

2.2 Lemmas 

Assume there exists an M-curve Cg with real scheme {J II l(/3 II 1(7))). 

We repeat hereafter the arguments from [6J for the case (3 = 0: let 
A, B, C be 3 inner ovals. The pencils of lines J- a ■ B — > C, : C — > A, 
and .Fc : A — > £? have no ^7-jumps and give rise to a closed Fiedler chain 
involving all of the inner ovals. Therefore, A+ — A_ = 0, II + — II = ±1 
and A + — A_ € {0, 2, —2}. This contradicts the Rokhlin-Mishachev formula. 
The real scheme {J II 1(1(26))) is not realizable. 

Let now (3 > 0. Assume Cg has Oi-jumps. Let e±, 62 and ne^ be respec- 
tively the contributions of 0±, O2 and of the inner ovals to A + — A_ (where 
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ei,e 2 ,e 3 G {+1,-1}, and n is non-negative). By Lemma 1 (4) and the 
Rokhlin-Mishachev formula, one must have: 2(— eie 2 — neie 3 ) + ei + e 2 = 8. 
There are 4 possible solutions: 

ei = e 2 = -1, e 3 = 1, n = 6; 

ei = -1, e 2 = e 3 = 1, n = 3; 

ei = 1, e 2 = e 3 = — 1, n = 3; 

ei = e 2 = 1, e 3 = -1, n = 4. 

In the second case, one has Z+ = 1, 7rl — 7r+ = 0; in the third case, 
one has l + = 1, tt± — 7r+ = 3. This contradicts the first Orevkov formula. 
Either of the other 2 cases verifies both Orevkov formulas. The numbers n, 
(3, 7 have the same parity. If Cg has Oi-jumps, these numbers are even, and 
Theorem 2 (1) is proven. 

Lemma 2 Let Cg be an M-curve with real scheme (J7II1(/3II1(7}}} ; without 
Oi-jumps and such that 7 > 2 and (3 > 1. Let A be a median oval, and B, C 
be the extreme inner ovals met by the pencil Ta sweeping out 0\. The 
complete pencil Tc gives rise to a cyclic Fiedler chain involving all other 
empty ovals. In the corresponding cyclic ordering, all of the inner ovals are 
consecutive. 

Proof: Let T ± , . . . , T 4 be the 4 triangles ABC, T x and T 2 being the 2 ones 
with edge [BC\. The inner ovals lie in Z[ U Z' 2 , where Z[ = T { n Int(Oi), 
i = 1,2. One of the zones = 1,2 is empty. Indeed, assume either zone 
contains an oval Ei. Then the conic through A, B, C, E\,E2 cuts Cg in more 
than 18 points. Contradiction. Consider the pencil of lines Tc starting at B 
and sweeping out the non-empty zone Z[, i € {1, 2}. Let D be the last inner 
oval met by this pencil. The pencil Tc ■ B — > D meets no median oval. 
Indeed, assume that a median oval E is met by this pencil. Then, the conic 
EBADC cuts Cg in more than 18 points. Contradiction. The complete 
pencil Tc gives rise to a cyclic Fiedler chain involving all other empty ovals. 
This chain splits into 2 consecutive subchains formed respectively by the 
inner and the median ovals. □ 

Lemma 3 Let Cg be an M-curve of degree 9 with deep nest, and 1 be any 
one of the inner ovals. Assume there exist five other empty ovals 2, 3, 4, 5, 6 
met successively by the pencil of lines T^ , and such that there is a J -jump 
between any two successive ovals in the cyclic ordering. Denote by ^11^456 
the pencil of rational cubics through 1, . . . , 6 with double point at 1. Then, 
up to cyclic permutation of 2, ... ,6, the sequence of singular (i.e. reducible) 

Cubics of ^1123456 
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16 U 14523, 14 U 12356, 12 U 14365, 15 U 12643, 13 U 15426 or 
12 U 14365, 15 U 12643, 16 U 12543, 13 U 12456, 14 U 12356 or 
16 U 15234, 14 U 15326, 15 U 13264, 13 U 14265, 12 U 14365 



See Figures 9, 10, 11. 

Proof: Notice first that if T\ : X, Y, Z, X has a ^7-jump between X and 
Y and between Y and Z, then: T\ has also a ^7-jump between Z and X, and 

1 lies in the principal triangle XYZ. Thus, 1 G 234n345n456n562n623. As 
no three points among 1, . . . , 6 can be on a line, these five principal triangles 
must have a 2-dimensional intersection. 

Case 1: The five points lie in convex position. Consider a pair of points, 
that are consecutive for , say 2,3, and assume these points also consecu- 
tive for the convex cyclic ordering. This cyclic ordering is 2, 3, X, Y, Z, with 
X,Y,Z G {4,5,6}. If X,Y,Z = 6,5,4 or 5,6,4, then 234 n 345 = [34]; if 
X,Y,Z = 6, 4, 5 or 5, 4, 6, then 632 n 456 = 6; if X, Y, Z = 4, 6, 5 or 4, 5, 6, 
then 234 n 456 = 4. Contradiction. Then, the only possible convex cyclic 
ordering of the five points is 2, 4, 6, 3, 5. Choose a line at infinity L that does 
not cut the convex hull of the points. In the oriented affine plane MP 2 \ L, 
there are a priori two possibilities for the positive cyclic convex ordering of 
the five points: 2, 4, 6, 3, 5 and 2,5,3,6,4. As sweeps out successively 
2, 3, 4, 5, 6, the first of these possibilities is realized, (see Figure 3). 

Case 2: One of the points, say 2, lie in the convex hull of the other 
four. Assume 3 and 4 are consecutive in the cyclic convex ordering. Then, 
234 n 562 = 2. Contradiction. There are a priori two possibilities for the 
positive cyclic convex ordering of the four points: 3, 5, 4, 6 or 3, 6, 4, 5. In the 
first case, the quadrangle 3546 is divided into four triangles: T\ = 345 n 356, 
T 2 = 356 n 346, T 3 = 346 n 456 and T 4 = 345 n 456. If 2 G T x U T 2 , 
then 623 n 456 = 6; if 2 G T 3 , then 623 n 345 = 3. Contradiction. Hence, 

2 G T4. In the second case, the quadrangle 3645 is divided into four triangles: 
Ti = 346H356, T 2 = 356H345, T 3 = 345R456, T 4 = 346R456. If 2 G T1UT4, 
then 632H345 = 3; if 2 G T 2 , then 623H456 = 6. If 2 G T 3 , then 1 G 234H562 
and the pencil sweeps out successively 6, 5, 4, 3, 2. This is a contradiction. 
Thus, 2 G T 2 (see Figure 4). 

Case 3: Two of the points lie in the principal triangle determined by the 
other three. Assume the two points are not consecutive for J 7 ^ , say these 
points are 4,6. Then 234 n 345 = [34]. Contradiction. Thus, up to cyclic 
permutation of 2, 3, . . . , 6, one can choose 4, 5 as interior points. There 
are a priori two possibilities for the positive convex ordering of the other 
three points: 2, 6, 3 and 2, 3, 6. As 1 G 632 and sweeps out successively 
6, 2, 3, the latter possibility is excluded. The triangle 632 is divided in six 
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triangles T» , i = 1, . . . , 6 by the lines 42, 43 and 46, such that: T 1 UT 2 = 346, 
T 3 U T 4 = 234, T 5 U T 6 = 246; and T U T 6 have 4,6 as common vertices, 
T 2 ,T% have 4,3 as common vertices, T^,T^ have 4,2 as common vertices. If 
5 G Ti U T 2 U T 4 U T 5 , then 345 n 456 = [45]; if 5 G T 6 , then 456 n 562 = [56]. 
Contradiction. One has 5 G T 3 (see Figure 5). 

Perform a Cremona transformation cr : {xq\ x±; x 2 ) — > {x\x 2 ; xgx 2 ; xqx±) 
with base points 1,5,4. Let us denote the respective images of the lines 
14,15,45 by 5,4,1. For the other points, we shall use the same notation 
as before cr. After cr, consider the pencil of conies T\ 2 ^. This pencil has 
3 singular conies: the double lines 12 U 36, 13 U 26 and 16 U 23, they are 
shown in Figures 6, 7 and 8. The position of the pencil ^"1236 with respect 
to the base lines is uniquely determined, it is shown in the left-hand side of 
Figures 9, 10, and 11. In cases 1, 2, this is obvious; in case 3, both points 
4, 5 are swept out in the portion 16 U 23 — > 13 U 26 and we have to find 
out which one is met first. The conic 16235 of T\ 2 m is the image of a conic 
15326 (see Figure 5), thus 4 lies outside of 16235. So, 5 is swept out before 4 
by ^1236- Perform the Cremona transformation back. The pencil of conies 
^1236 is mapped onto the pencil of rational cubics ^1123456 (right-hand side 
of Figures 9, 10, 11). □ 

Lemma 4 Let Cg be an M -curve of degree 9 with deep nest, and denote by 
1 any one of the inner ovals. The complete pencil of lines T\ has at most 3 
J' -jumps over the sequence of median ovals. 

Proof Assume there exists an inner oval 1 of Cg such that T\ has 5 
j7-jumps over the median ovals. Let 2, . . . , 6 be median ovals such that 
1, 2, . . . , 6 verify the conditions of Lemma 3. Notice that in the affine plane 
of Figures 9, 10, 11, a segment of cubic connecting two points among 1, . . . , 6 
goes over infinity if and only if it cuts J. In each of the 3 cases, the 
cubics of the pencil ^"1123456 intersect all 0\ U 2 U {1, 2, 3, 4, 5, 6} U J at 27 
points, so there is no possibility to sweep out the other empty ovals of Cg. 
Contradiction. □ 

2.3 Proof of Theorems 1 and 2 

Let Cg be an M-curve of degree 9 with real scheme {J II l(/3 II 1(7}}) and 
without Oi-jumps. Let e\,e 2 and ne% be respectively the contributions of 
0\, 2 and of the median ovals to A + — A_; if 7 is odd, let e 4 be the 
contribution of the inner ovals to A + — A_ (where €1,62,63, 64 G {+1, — 1} 
and n is non-negative). Let A be a median oval and C be one of the extreme 



9 



Figure 3: Case 1 




Figure 6: The 3 double lines of T\im-, case 1 




16 U 23 




Fi gure 11: ^1123456 



16 U 15234 




= cr H-^me), case 3 



inner ovals met by Ta- Applying Lemmas 2 and 4 with the complete pencil 
of lines J-q, we prove that if 7 is odd, then n G {1>3} and if 7 is even, 
n G {0,2,4}. The Rokhlin-Mishachev formula yields respectively for odd 
and for even 7: 

2(— eie 2 - e 4 e 2 - e 4 ei - ne 3 e 2 ) + e\ + e 2 + ne 3 + e 4 = 8 

2(— eie 2 - ne 3 e 2 ) + e\ + e 2 + ne 3 = 8 

For 7 odd, there is no solution. For 7 even, there are 2 solutions: 
ei = e 2 = -1, e 3 = 1, n = 4; 
ei = e 3 = 1, e 2 = -1, n = 2. 

In the latter case, one has: l + = 1, irt — 7r+ = 0. Contradiction with the 
first Orevkov formula. In the first case, both Orevkov formulas are verified. 
The numbers n, (3, 7 have the same parity, if Cg has no Oi-jump, these 
numbers are even. This finishes the proof of Theorem 1 and of Theorem 2 
(2). □ 
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